A Perceptually-Motivated Optimization-Framework for Image and Video Processing

We present an optimization framework for expressing image processing applications that can account for certain perceptual biases of the human vision system (HVS). Perception literature is ripe with studies demonstrating the HVS to be more sensitive to pixel gradients than absolute pixel values, which has led to some important work in gradient domain image filtering. Inspired by this work, our optimization framework allows image and video processing applications to easily specify both zeroth order constraints (i.e., desired pixel values) and first order constraints (i.e., desired pixel gradients in space and time) in the optimization. We introduce a spatiallyvarying weighting scheme for these constraints that produces perceptually pleasing results by approximating the more robust L1norm even when using a simple weighted least squares optimization. We also demonstrate that edge length in addition to local gradient magnitude is a useful measure of local gradient saliency. Our saliency measure is inspired by perception studies that show long coherent edges in an image, even when faint, are perceptually salient to the HVS. Finally, we demonstrate the utility of our formulation in creating effective yet simple to implement solutions for common image processing tasks. To exercise our formulation we have created a new saliency-based sharpen filter and a pseudo image relighting application. We also revisit and improve upon filters previously defined by the gradient domain community – filters like painterly rendering, image de-blocking, and sparse data interpolation over images (e.g., colorization using optimization).